Dynamic adjustment of image compression for high resolution live medical image sharing

ABSTRACT

A video stream of live medical images is generated at a local site having a medical image scanner. A live video stream is transmitted to at least one remote site via a network, which may include wired or wireless Internet connections. Network conditions are monitored during a network session and predictions are made on a predicted bit rate for transmission. The compression parameters for the live video stream are selected based on the predicted bit rate.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of U.S. ProvisionalApplication No. 61/829,887, filed on May 31, 2013, the contents of whichare hereby incorporated by reference.

FIELD OF THE INVENTION

The present invention is generally related to techniques for livesharing over the Internet of a video stream composed of high resolutionmedical images, such as ultrasound images. More particularly, thepresent invention is directly to dynamically adapting compressiontechniques for a video stream of medical images to adjust to variablenetwork conditions.

BACKGROUND OF THE INVENTION

In telemedicine applications there is a need to share medical databetween different sites. However, one problem in the industry is thatsome medical imaging applications require a large bandwidth to stream asa live video stream of medical images. Illustrative examples includelive video streaming of ultrasound imaging, angiography, and endoscopy.Many types of medical images are difficult to efficiently compress. Forexample, ultrasound images have a high entropy content and havecompression ratios that are dramatically lower than the compressionratios that can be achieved for streaming television and movies. Anadditional complication is that many small clinics have poor ITinfrastructure with marginal and time-varying connections to theInternet, due to cost considerations, remote locations, or otherreasons.

As an illustrative example of some of these problems, in ultrasound(u/s) imaging, image frames typically have a resolution of 512×512pixels that are acquired at frame rates of 10 to 60 frames per second(fps). The frame rate may vary, depending in the u/s frequency ofoperation. The frame size is usually fixed, and the pixel resolution isagain determined by the u/s frequency, and can vary from 400 microns at2 MHz frequency to 80 microns at 10 MHz frequency. It is this pixelresolution that is relevant to the end-user, not the frame size itself.The frames at the frame rate are usually concatenated to form a standardvideo stream.

The stated frame size and rates mentioned above imply a raw data rate of63 Mbps at 30 fps and 8 bits per pixel, gray scale. Without anycompression, the data storage required to store one minute of the u/sstream at 30 fps and 8 bits per pixel is 450 MB. For color Doppler u/simaging 12 bits per pixel is required, implying a raw data rate of about95 Mbps at 30 fps and a data storage requirement to store one minute ofthe Doppler u/stream of 675 MB. Consequently, an image compressionscheme has to be employed to reduce the bit rate for real-time networktransport of the u/s scheme and to reduce the data storage requirementsof the u/s scheme. However, the compression requirements for these twouse cases need not be identical.

In medical imaging, lossless and lossy compression schemes are employed,depending on the imaging modality. Lossy compression may be employed aslong as the quality of compressed images does not violate the JustNoticeable Difference (JND) threshold. This threshold is verysubjective, but, many standards bodies such as the American College ofRadiology (ACR) have established guidelines for compression rates forvarious medical imaging modalities.

Ultrasound images have a high entropy content and cannot be compressedwith as high a compression ratio as conventional video streams formovies. For ultrasound imaging, many different compression standards areallowed. Examples of permissible compression standards for ultrasoundimaging include Motion JPEG2000 (MJPEG2000), MPEG-4 and H.264.

Motion JPEG is a video codec in which each frame is separatelycompressed into a JPEG image. As a result the quality of the videocompression is independent of the motion of the image. At low bandwidthavailability priority is given to image resolution. In contrast, MPEG-4is a standard that sends a reference frame and difference data forfollowing frames (I frames, B frames, and P frames).

In the case of MJPEG2000, each image frame is compressed (either lossyor lossless) and every frame in the 30 fps stream is sent separately.Typical lossy compression rates (compression ratios) are of the order of1:10 to 1:15. Further effective compression rates are not possiblewithout losing image quality, since inter-frame data redundancy is notcaptured in MJPEG2000. Thus, while the compression rates of 1:10 to 1:15rates of MJPEG2000 are good, there are problems in using MGPEG2000 fordata storage or for network transport.

In the case of MPEG-4 and H.264, larger compression rates, from 1:20through 1:80, are possible, since they utilize frame-to-frameredundancies and motion vector compensation schemes. For high entropycontent images, such as ultrasound, the compression rates on the orderof up to 1:20 to 1:40 are possible. These compression standards alsoallow for segmenting the images into multiple slices, and applydifferent compression rates for different schemes. An ultrasound imagetypically includes a main ultrasound image 105 (the active sub-imagetaken by the ultrasound probe) and border regions 110, 115 which mayinclude labeling or text describing aspects of the image. Thus an imagecan often be segmented into strips. For the image example in FIG. 1, theimage can be segmented into three strips—a top stripe having sometextual data, a left stripe having textual data and the remainingsub-image which corresponds to the ultrasound image data.

The top and left stripes are highly compressible, to a few bytes, sinceinter-frame redundancies are very high. The active sub-image will have alower compression rate, based on the mobility of the organ under exam.

Image data may be sent via a wired or wireless network. In a networkedenvironment such as the internet, where these u/s streams aretransported in real-time, there are dynamic conditions of the networkthat can momentarily constrict or disrupt the available bandwidth foru/s stream transport. As a result there can be a severe loss in thequality of the real time transaction and/or a loss in the connection,which is unacceptable in many applications.

SUMMARY OF THE INVENTION

An apparatus, system, method and computer readable medium is disclosedto dynamically adjust compression parameters for a live video stream ofmedical imaging data in a network having variable network conditions.Quality of Service (QOS) metrics are monitored for a network sessionbetween at least two sites. In one embodiment the QOS metrics are usedto predict a minimum expected bit rate. The minimum expected bit rate isused to select compression parameters for compressing the live videostream of medical imaging data. The compression parameter may includedifferent video compression protocols and selectable parameters withinan individual compression protocol. In one embodiment the quality ofservice metrics are also used to select a network transmission protocolfor transmitting the video stream.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a conventional ultrasound image with dashed linessuperimposed to illustrate aspects of image compression.

FIG. 2 illustrates a system for dynamically adjusting compressionparameters of a video stream of medical imaging data in accordance withan embodiment of the present invention.

FIG. 3 illustrates a tunable compression engine in accordance with anembodiment of the present invention.

FIG. 4 illustrates aspects of a network QOS transaction in accordancewith an embodiment of the present invention.

FIG. 5 illustrates prediction of compression rate in accordance with anembodiment of the present invention.

FIG. 6 illustrates selection of network transmission protocol based onnetwork conditions in accordance with an embodiment of the presentinvention.

DETAILED DESCRIPTION

FIG. 2 illustrates an exemplary system and network environment forsharing a video stream of compressed medical images. At a local clinicsite 205 a patient 207 is examined by a doctor 209 or a medicaltechnician. The patient may by a human patient. Alternatively, manymedical imaging procedures have been adapted for veterinary medicinesuch that the patient may be cat, dog, horse, etc. A medical imagingscanning device 210 generates a live stream of video images that aretransmitted over a network to another site 260, such as local areanetwork 265 of a medical center. As an illustrative example, the livestream may be transmitted to a site of a specialist doctor or a doctorfrom whom a second opinion is desired. It is also understood that thelive stream may also be transmitted simultaneously to other sites.

An exemplary medical imaging scanning device 210 is an ultrasoundimaging device, although more generally other types of live imagingdevice could be used, such as angiography or endoscopy. For the case ofultrasound there is high entropy content of the images in the videostream which in turn invokes many tradeoffs in regards to thecompression parameters used to compress the images. Exemplary imagingtechnologies may require frame rates of 10-60 fps, 8 bits per pixel grayscale and 12 bits for color images, such as color Doppler ultrasoundimages. In the case of ultrasound imaging, image frames may have aresolution of 512×512 pixels at frame rates of 30 fps and 8 bits perpixel, the raw data rate is 63 Mbps. Other medical imaging techniques,such as angiography, have similar data requirements.

The present invention is generally directed to the use of dynamicallyadapting a compression scheme used for transporting a medical imagevideo stream, such as an ultrasound stream, across a network with timevarying network session connection quality using feedback from at leastone network quality of service agent. A live stream of medical imagescarries a large amount of data. Additionally, certain types of medicaldata, such as ultrasound images, are difficult to efficiently compress.Moreover, the network connection quality between sites may varydramatically. For example, a doctor at a small clinic may have a poorquality connection to the Internet. As a result of these factors,compressing the live video stream of the medical imaging data isperformed to attempt to maintain live connection with minimaldegradation of the quality of the live images.

The network path to a remote viewer at site 260 includes the Internetnetwork cloud 250 and any local networks, such as local network 265.Reporting (R) tools are network agents that provide network metrics atdifferent part of the network communication path. Typically there wouldbe reporting tools configured in at least both ends of the network path.These network metrics may include attributes such as bandwidth, packetloss, and packet corruption. The reporting tools may comprise commercialor proprietary reporting tools. The frequency with which reports arereceived may be configured. For example, many commercial networkreporting tools permit periodic generation of reports on networkconditions such as once every 100 ms, once every second, once every fiveseconds, etc.

The network quality of service (QOS) metrics are monitored and used topredict network conditions (in the near future) to determine optimumcompression parameters for transmitting a live video stream of medicalimages to the remote viewer. That is, the QOS metrics provide metrics onpast and recent network conditions, which are then used to predictnetwork conditions when a frame of the live video stream is transmitted.For example, suppose the reporting tools provide a report every 5seconds. In this example, if the last report was received 3 seconds ago,data on the past and most recent report (3 seconds ago) on networkconditions may be used to predict network conditions to transmit a videoframe. In particular, an expected minimum bit rate may be calculatedbased on quality of service inputs such as predicted bandwidth,predicted packet loss, and predicted packet latency. This minimum bitrate of the connection session, in turn, implies a compression rate, orcompression ratio, for the live stream of medical images.

Block 240 illustrates an example of modules to dynamically adjustcompression parameters as network conditions vary. In one embodiment alocal computer 249 includes software modules to perform network QOSmonitoring 242, compression rate prediction 244, a tunable compressionengine 246, and a call management selection module 248. In oneembodiment the call management selection module 248 also receives theQOS metrics from network QOS monitoring module 242 and selects thenetwork transmission protocol (e.g., TCP or UDP) based on the networkconditions.

The tunable compression engine 244 has compression parameters that areselectable. The selectable parameters may include different compressionprotocols and/or selectable features within one protocol. The tunablecompression engine selects the compression technique, based on thepredicted compression rate, to optimize the image quality for themedical images of the live video stream. In particular, the tunablecompression engine may select the compression technique to minimizedistortion of the video images given the constraint of the predicted bitrate and that certain types of medical images, such as ultrasoundimages, have a high entropy content. The compressed live medical imagestream is transmitted using a network protocol, which may also beadjusted based on network conditions.

FIG. 3 is a functional block diagram illustrating additional aspects ofthe compression engine 246. A rule-based system may select an optimumcompression technique to transmit a live video stream of medical imageswith minimal distortion. That is, certain ranges of parameters, such asbandwidth, loss, and packet corruption, are mapped to specificcompression parameters. The compression engine in one embodiment selectsa compression protocol from a choice of at least two differentcompression protocols used in transmitting live video streams of medicalimages. The compression engine may also make dynamic adjustments toindividual selectable features of an individual compression protocol.The compression engine is a flexible engine. Depending on the availablebandwidth (predicted bit rate), it can switch dynamically betweendifferent video compression protocols. In one embodiment it candynamically switch between MJPEG2000 compression and standardMPEG-4/H.264 schemes. More generally, it is contemplated that othercurrently implemented and proposed compression standards may beselected. Other examples of video compression protocol standards thatmay be selected include H.265, VP8, and VP9. Additionally, in oneembodiment at least one compression protocol has selectable featuresthat can be turned on or off. In the case of MPEG-4 and H.264,compression rates are adjusted by one or more of:

1) turning on or off MVC encoding,

2) varying the block sizes of frames,

3) varying the quantization tables for intra-frame discrete cosinetransform (DCT) compression,

4) varying motion vector compensation (MVC) for inter-frame compression,and

5) reducing the stream frame rate by dropping frames ahead of thecompression engine.

As also illustrated in FIG. 3, in one embodiment an additional storagecompression engine 305 may be provided to optimize compression forstorage. In some applications it may be desirable to also store thevideo stream. Consequently, as illustrated in FIG. 3, a parallelcompression engine for storage of the compressed medical image videostream may be provided that is not subject to such dynamic compressionschemes. The compression engine for storage may use a constantconservative compression rate applied to the stream that will notcompromise the JND threshold guidelines of ACR.

Note that the compression engine can adapt rapidly to changing networkconditions. In many parts of the world doctor's offices and smallclinics may use wireless local networks and/or wireless Internetconnections. However, the quality of these connections may depend on thetime of day and other factors. For example, a clinic close to a traintrack may experience large changes in the quality of a wireless Internetconnection whenever a train travels close to the clinic, due to thereflection from the metal surface of the train. As another example, forthe case of a small clinic using a WiFi connection, the quality of WiFiservice may vary based on time of day and number of users. Wirelessinternet connections based on the 802.11 standard may sufferinterference and packet drop at times of the day when there are manyusers on individual wireless networks and on neighboring wirelessnetworks. As an example, many wireless internet systems in individualportions of a city slow down between the hours of 5 PM to 8 PM asmultiple users in the same geographic region attempt to simultaneouslyaccess the Internet.

The compression engine 246 compresses the video stream prior totransmission. In practice the network has time-varying quality ofservice (QOS) characteristics, such as packet loss, bandwidth, andpacket corruption. If fixed (non-varying) compression techniques areused then in the worst case the changing network conditions can resultin a severe loss in the quality of the real time transaction and apossible loss in connection, which in unacceptable. In accordance withan embodiment of the present invention, the compression engine reacts tosuch changing conditions by monitoring network conditions and adaptingthe compression technique used to compress the video stream. Thecompression scheme is adapted to the dynamic conditions of the networkby utilizing a variable rate compression scheme. For the data archivaluse-case, a fixed rate compression scheme is adequate and the rate isbased on the JND metric for the specimen being imaged.

FIG. 4 illustrates aspects related to the use of Active Network QualityAgents (ANQA) for an ultrasound (u/s) embodiment. The ANQA elements 405serve as the reporting (R) agents. In a network session the network canbe pictured as a network pipe 402 between local and remote sites. TheANQA elements may be based on proprietary software or on commercialsolutions. As an example, commercial products, such as the Airwavesoftware of Aruba Networks, Inc., provide network Quality of Service(QOS) metrics, and can be configured to report these statistics todesired locations. The ANQA elements may be local and remote reportingelements, although more generally there may be additional ANQA elementsat different points in the network. Examples of QOS metrics includecurrent bandwidth (BW) for TCP connections and packet loss rate (PLR)for UDP connections

In one embodiment, the Active Network Quality Agent (ANQA) continuallymonitors the available QOS metrics for a network session. In oneembodiment a prediction scheme for determining the available bandwidthis utilized. That, is the current and recent reporting results from theANQA agents are used to predict the expected bit rate along thetransmission path when the video is transmitted.

The ANQA reporting agent(s) 405 report the QoS metrics to theCompression Rate Predictor (CRP) 410, which outputs a compression rateto the compression engine 420. The CRP may rely on available metrics forcurrent and immediate past of network Quality of Service (QoS) topredict a compression rate which would be compatible with the networkconditions.

The CRP may, for example, utilize linear or non-linear predictiontechniques for compression rate prediction. That is, a variety ofdifferent well-known prediction algorithms may be modified to take theQOS metrics (e.g., bandwidth, packet loss, and packet corruption) andgenerate an expected bit rate for the compression engine.

A variety of linear prediction algorithms may, for example, predictnetwork quality (e.g., packet loss, bandwidth) based on regressiontechniques and, curve fitting based on the measured QOS metrics topredict network characteristics. Referring to FIG. 5, in one embodiment,a filter is provided in combination with a forward predictor. Thecurrent compression rate is an input and the output is a predicatedcompression rate. In one embodiment a Linear Prediction algorithmutilizes Auto-Regressive Moving-Average (ARMA) models that utilize anerror correction scheme such that the filter would be an ARMA filter.Kalman Prediction (KP) may also be used by the compression ratepredictor. As examples, any KP scheme (Linear or Non-Linear) that ismodeled in a higher order state space scheme may be used. Additionally,Hidden Markov Model (HMM) Prediction may be used by the compression ratepredictor.

In one embodiment the monitoring rate of ANQA agents is adjustable. Forexample, this monitoring can be done every second to about few times aminute. This monitoring rate can also be subject to a prediction scheme,based on the rate of change of QoS, in a manner similar to thecompression rate prediction.

FIG. 6 illustrates an example for the medical image scanner being anultrasound imager 610. Referring to FIG. 6, in one embodiment the ANQAare used to support a call management system dynamically switchingbetween different network transmission protocols (e.g., TransmissionControl Protocol (TCP) and User Datagram Protocol (UDP) protocols) basedon QoS metrics. To monitor an individual session, at least one ANQAagent is monitored. Preferably at least two ANQA agents (one at eitherend of the network connection is monitored). Additional ANQA agents inthe cloud may also be optionally monitored. There are tradeoffs, interms of bandwidth and packet loss for different network protocols. TCP,for example, provides for retransmission of dropped packets and thusrequires higher bandwidth than UDP. UDP has lower bandwidth and lowerlatency requirements but does not guarantee delivery of packets.

Additionally, as previously discussed, the QOS metrics may differentslightly for TCP vs. UDP and thus have to be factored into thecompression rate predictor. Thus, the network transmission protocol maybe selected to optimize the quality of the live video stream in view ofthe QOS metrics. While TCP and UDP are examples of network transmissionprotocols, it will be noted that other network transmission protocolsare contemplated, such as Stream Control Transmission Protocol (SCTP).

In one embodiment the QOS statistics during use of the medical imagescanner may also be monitored by a service entity to providerecommendations. The QOS statistics provide information on the networkconnection quality. The QOS statistics may also be collected over manydifferent sessions. As a result, temporal patterns may be detected. Alarge hospital or clinic may have a dedicated in-house informationtechnology capability. However, small clinics or individual doctorsoften lack such capabilities and under-invest in information technologyservices. In one embodiment a rule based system may providerecommendations on how an upgrade in network capabilities (e.g.,connection type and/or connection bandwidth) may improve performance.

It will be understood that the networks metrics monitoring, compressionrate predictor, tunable compression engine, and call management may beimplemented in different ways. In one embodiment they are implemented assoftware modules operating on a local computer. Alternatively, it willbe understood that some or all of the modules may be implemented withina medical image scanner. It will be understood that the software modulesand methodologies may be stored as computer readable instructions storedon a non-transitory computer readable medium.

While the invention has been described in conjunction with specificembodiments, it will be understood that it is not intended to limit theinvention to the described embodiments. On the contrary, it is intendedto cover alternatives, modifications, and equivalents as may be includedwithin the spirit and scope of the invention as defined by the appendedclaims. The present invention may be practiced without some or all ofthese specific details. In addition, well known features may not havebeen described in detail to avoid unnecessarily obscuring the invention.In accordance with the present invention, the components, process steps,and/or data structures may be implemented using various types ofoperating systems, programming languages, computing platforms, computerprograms, and/or general purpose machines. In addition, those ofordinary skill in the art will recognize that devices of a less generalpurpose nature, such as hardwired devices, field programmable gatearrays (FPGAs), application specific integrated circuits (ASICs), or thelike, may also be used without departing from the scope and spirit ofthe inventive concepts disclosed herein. The present invention may alsobe tangibly embodied as a set of computer instructions stored on acomputer readable medium, such as a memory device.

What is claimed is:
 1. A method of transmitting over the Internet a livevideo stream of ultrasound imaging data, comprising: monitoring a set ofquality of service metrics for a network communication session between alocal site having an ultrasound imaging scanner and a remote site;predicting, based on the set of quality of service metrics, a minimumexpected bit rate to transmit a compressed video stream including a liveultrasound video stream; selecting, in response to the expected bitrate, video compression parameters to compress the video streamincluding the live ultrasound video stream; and transmitting thecompressed video stream to the remote site.
 2. The method of claim 1,wherein the at least one quality of service metric comprises a bandwidthmetric, a packet loss metric, a packet latency, and a packet corruptionmetric.
 3. The method of claim 1, wherein the predicting comprises aprediction technique selected from the group consisting of a linearpredication algorithm and a nonlinear prediction algorithm.
 4. Themethod of claim 1, wherein the predicting comprises a predictiontechnique selected from the group consisting of a linear predictionalgorithm, Kalman prediction, and hidden Markov model prediction.
 5. Themethod of claim 1, wherein the live medical image video stream comprisesone of an ultrasound video stream, an angiography video stream, and anendoscopy video stream.
 6. The method of claim 1, wherein the selectingcomprises selecting from at least two different video compressionprotocols.
 7. The method of claim 6, wherein the at least two differentvideo compression protocol include a MJPEG2000 compression protocol andat least one of MPEG-4, H.264, H.265, VP8, and VP9.
 8. The method ofclaim 1, wherein the selecting comprises selecting a subset of featuresof a particular video compression protocol.
 9. The method of claim 8,wherein the selecting a subset of features includes selecting at leastone of: turning on or off the MVC encoding; varying the block sizes offrames; varying the quantization tables for intra-frame discrete cosinetransform (DCT) compression; varying motion vector compensation (MVC)for inter-frame compression, and reducing the stream frame rate bydropping frames ahead of the compression engine.
 10. The method of claim1, further comprising selecting a transmission protocol based on theexpected bit rate.
 11. The method of claim 1, further comprisinganalyzing quality of service metrics for a plurality of sessions andproviding a network service recommendation.
 12. A method of tuning videocompression parameters for a live video stream of medical images in anetworked telemedicine environment having variable network quality,comprising: monitoring at least one quality of service metric for anetwork session between a local site having a medical imaging scannerand a remote site; predicting, based on the at least one quality ofservice metric, a minimum expected bit rate to transmit a compressedlive video stream of high resolution medical images from the medicalimaging scanner; selecting, in response to the expected bit rate, videocompression parameters for the live video stream of medical images;compressing the live video stream of high resolution medical image,using the selected video compression parameters; and transmitting thecompressed live video stream to the remote site.
 13. The method of claim12, wherein the at least one quality of service metric is monitored inat least a local site and remote site.
 14. The method of claim 12,wherein the at least one quality of service metric comprises a bandwidthmetric, a packet loss metric, a packet latency, and a packet corruptionmetric.
 15. The method of claim 12, wherein the predicting comprises aprediction technique selected from the group consisting of a linearpredication algorithm and a nonlinear prediction algorithm.
 16. Themethod of claim 12, wherein the predicting comprises a predictiontechnique selected from the group consisting of a set of linearprediction algorithms, Kalman prediction, and hidden Markov modelprediction.
 17. The method of claim 12, wherein the live medical imagevideo stream comprises one of an ultrasound video stream, an angiographyvideo stream, and an endoscopy video stream.
 18. The method of claim 12,wherein the selecting comprises selecting from at least two differentvideo compression protocols.
 19. The method of claim 18, wherein the atleast two different video compression protocol include a MJPEG2000compression protocol and at least one of MPEG-4, H.264, H.265, VP8, andVP9.
 20. The method of claim 12, wherein the selecting comprisesselecting a subset of features of a particular video compressionprotocol.
 21. The method of claim 20, wherein the selecting a subset offeatures includes selecting at least one of: turning on or off the MVCencoding; varying the block sizes of frames; varying the quantizationtables for intra-frame discrete cosine transform (DCT) compression;varying motion vector compensation (MVC) for inter-frame compression,and reducing the stream frame rate by dropping frames ahead of thecompression engine.
 22. The method of claim 21, further comprisingselecting a transmission protocol based on the expected bit rate. 23.The method of claim 12, further comprising analyzing quality of servicemetrics for a plurality of sessions and providing a network servicerecommendation.
 24. A non-transitory computer readable medium includinginstructions which when executed on a processor implement any one of themethods of claim 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, or
 23. 25.A system, comprising: an ultrasound image scanner to generate a livevideo stream of ultrasonic medical images; and a computer system tomonitor network conditions in a network session to a remote site,predict a minimum bit rate, and select compression parameters tocompress the live video stream prior to transmission to the remote site.